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FOREWORD 


This  research  and  development  was  conducted  under  the  sponsorship  of  the  Defense 
Advanced  Research  Projects  Agency  and  is  related  to  studies  of  criterion-referenced 
testing  being  conducted  at  this  Center.  Information  resulting  from  this  testing  will  be 
incorporated  in  a  testing  manual  being  prepared  by  the  Navy  Personnel  Research  and 
Development  Center.  This  manual  will  be  used  operationally  by  the  Chief  of  Naval 
Education  and  Training,  the  Chief  of  Naval  Technical  Training,  and  the  Chief  of  Naval 
Education  and  Training  Support  (specifically,  the  Instructional  Program  Development 
Centers). 

A  previous  report,  NPRDC  TR  78-23  of  June  1978,  described  the  beginning  phases  of 
a  contractual  effort  aimed  at  examining  the  qualities  of  test  questions  written  from  a 
variety  of  methods.  This  report  describes  a  replication  and  extension  of  that  work. 
Results  will  be  considered  in  further  development  of  algorithmic  procedures  for  generat¬ 
ing  test  questions  from  prose  materials. 

Appreciation  is  expressed  to  Dr.  John  R.  Dormuth  of  the  University  of  Chicago,  and 
Dr.  Jason  Millman  of  Cornell  University,  who  were  consultants  for  this  project. 

Dr.  Pat-Anthony  Federico  of  this  Center  served  as  the  Contracting  Officer  Technical 
Representative. 


DONALD  F.  PARKER 
Commanding  Officer 
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SUMMARY 


Problem  and  Background 

Methods  lor  writing  test  questions  or  items,  particularly  for  criterion-referenced 
testing,  are  needed  that  are  (1)  based  on  a  logically  defined  relationship  between  the 
instructional  materials  and  the  test  items  written  to  assess  learning  from  those  materials, 
and  (2)  capable  of  producing  items  that  can  be  easily  replicated  by  many  test  developers. 
Such  methods  should  allow  tests  to  become  more  scientific  instruments  and  contribute  to 
the  advancement  of  instructional  research,  educational  evaluation,  and  the  use  of  test 
data  in  forming  public  policy. 

In  ari  earlier  study  (NPRDC  TR  78-23),  an  attempt  was  made  to  refine  a  method  of 
objectively  generating  multiple-choice  test  questions  by  transforming  sentences  from 
prose  instructional  materials  and  developing  foils  or  question  alternatives  by  an  al¬ 
gorithmic  method.  In  that  study,  selected  instructional  material  was  computer-analyzed 
to  identify  high  information  words--those  that  are  relatively  rare  in  American  Eng- 
lish--and  to  determine  the  text  frequency  of  those  words.  Twenty  high  information  nouns 
ana  adjectives-- 10  rare  singletons  and  10  keywords--were  selected  for  use  as  questions 
words.  Singletons  are  high  information  words  that  occur  only  once  in  a  passage;  and 
keywords,  those  that  occur  more  than  once.  Twenty  sentences  were  then  selected  for 
transformation  into  items  by  four  item  writers.  Five  of  these  sentences  included  rare 
singleton  nouns;  five,  rare  singleton  adjectives;  five,  keyword  nouns;  and  five,  keyword 
adjectives. 

The  four  item  writers  transformed  the  selected  sentences  by  substituting  the  question 
words  with  wh-words  (who,  what,  etc.),  and  generated  item  foils  or  response  alternatives 
both  informally  and  with  an  algorithmic  method.  This  resulted  in  160  items--20  selected 
sentences  transformed  by  four  item  writers  using  two  foil  methods--that  were  organized 
into  eight  20-item  test  forms.  These  test  forms  were  administered  to  24  subjects-- three 
to  each  form--before  (pretest)  and  after  (posttest)  they  studied  the  instructional 
material.  Care  was  taken  to  ensure  that  students  completed  different  test  forms  on  the 
two  test  occasions.  Average  pretest  and  posttest  item  difficulty,  as  determined  by  the 
percentage  of  subjects  who  answered  the  question  correctly,  were  computed  for  items  (1) 
produced  by  each  of  the  four  writers,  (2)  derived  from  each  of  the  four  types  of  question 
words,  and  (3)  with  foils  generated  by  each  of  the  two  methods. 

Results  indicated  that  rare  singleton  nouns  and  adjectives  and  keyword  adjectives  are 
promising  candidates  for  use  as  question  words  in  developing  questions  that  test  learning 
from  prose.  Keyword  nouns,  however,  are  not  good  candidates.  It  was  concluded  that  the 
methods  used  to  generate  foils  algorithmically  were  feasible.  Although  foils  produced  by 
these  methods  were  somewhat  easier  than  those  generated  by  item  writers,  they  still 
appeared  to  produce  a  significant  shift  in  difficulty  from  pretest  to  posttest  when 
instruction  was  provided  between  testing  sessions. 

Purpose 


The  purpose  of  this  study  was  to  extend  or  replicate  the  earlier  study.  It  is  expected 
that  the  results  will  form  the  basis  for  additional  development  of  algorithmic  procedures 
for  generating  test  questions  from  prose  materials. 


vii 


Approach 


The  eight  forms  were  administered  to  249  high  school  students  before  and  after  they 
had  studied  the  instructional  material.  For  both  pre-  and  posttest,  about  30  students  were 
randomly  assigned  to  each  of  the  test  forms.  Care  was  taken,  however,  to  ensure  that  the 
forms  administeted  to  each  subject  on  the  two  test  occasions  were  different. 

To  obtain  stable  estimates  of  item  difficulty,  test  results  from  the  earlier  study  were 
combined  with  those  obtained  in  this  study.  Thus,  the  total  number  of  subjects  was  273 
(24  college  students  and  249  high  school  students).  A  repeated  measures  analysis  of 
variance  was  used  to  examine  differences  in  item  difficulties  between  (1)  the  four  item 
writers,  (2)  the  two  parts  of  speech  of  question  words,  (3)  the  two  types  of  text 
frequencies  (keyword  and  rare  singletons),  (4)  the  two  foil  types,  and  (5)  the  two  test 
occasions. 

Results 


1.  Items  based  on  rare  singleton  nouns  and  adjectives  and  keyword  adjectives 
showed  a  significant  change  in  item  difficulty  from  pretest  to  posttest,  indicating  that 
such  items  are  useful  in  learning  from  the  type  of  prose  used  in  the  study. 

2.  Items  derived  from  keyword  nouns  produced  low  quality  items,  primarily  because 
the  sentences  they  occurred  in  were  usually  introductory  sentences  of  a  general  nature. 

3.  The  two  types  of  foils  proved  to  be  almost  equally  effective  for  learning,  as 
evidenced  by  the  similarity  in  pcsuest  item  difficulty.  Those  generated  py  item  writers, 
however,  wore  considerably  harder  on  the  pretest  and  showed  a  higher  change  in  item 
difficulty  from  pretest  to  posttest  than  did  those  generated  algorithmically. 

4.  No  significant  differences  between  item  writers  were  found,  indicating  that  the 
sentence  transformation  methods  employed  apparently  neutralized  the  effects  of  item 
writer  bias  that  has  been  found  in  other  studies  of  item  writing. 

Conclusions 


The  concept  of  using  a  computer-based  algorithm  to  analyze  prose  instructional 
materials  and  to  identify  high  information  words  appears  to  be  workable.  High 
information  rare  singleton  nouns  or  adjectives,  as  well  as  keyword  adjectives  that  occur 
no  more  than  three  times,  appear  to  be  good  candidates  for  question  words.  Keyword 
nouns,  however,  apparently  are  not  good  candidates,  particularly  when  they  occur  in 
general  introductory  sentences. 

Recommendations 


1.  Rare  singleton  nouns  and  adjectives  and  keyword  adjectives  that  occur  infre¬ 
quently  in  instructional  material  should  be  used  to  select  sentences  from  prose  passages 
for  transformation  into  questions  that  measure  reading  comprehension.  Keyword  nouns 
should  not  be  u>ed,  particularly  when  they  occur  in  general  introductory  sentences. 

2.  Methods  of  algorithmically  generating  foils  for  multiple-choice  versions  of 
sentence-derived  questions  should  be  further  refined  and  applied  in  a  variety  of  subject 
matter  areas. 
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INTRODUCTION 


Problem 

Methods  for  wilting  test  questions  or  items,  particularly  for  criterion-referenced 
testing,  are  needed  that  are  (I)  based  on  a  logically  defined  relationship  between  the 
instructional  materials  and  the  test  items  wntten  to  assess  learning  from  those  materials, 
(2*  defined  by  a  set  of  operations  open  to  public  inspection,  and  O)  capable  of  producing 
items  tfut  tan  be  easily  replicated  by  many  test  developers.  Such  methods  should  allow 
tests  to  become  more  scientific  instruments  and  contribute  to  the  advancement  v ! 
instructional  research,  educational  evaluation,  and  the  use  of  test  data  in  forming  public 
policy. 

hov kg round 

Roid  and  Finn  (078)  attempted  to  refine  a  method  of  objectively  generating 
multiple-choice  test  questions  by  transforming  sentences  from  prose  instructional  mate¬ 
rials  and  developing  foils  or  question  alternatives  by  an  algorithmic  method.  A  prose 
passage  on  insect  development  (see  appendix),  which  was  written  for  approximately  the 
high  school  level,  was  selected  for  use  in  the  Roid  and  Finn  study.  Items  (stems  and  foils) 
to  test  learning  from  this  pa  . sage  were  developed  using  the  following  procedure: 

1.  The  selected  mat 'rial  was  computer-analyzed  to  identify  high  information 
words- -those  that  arc  re1  ,i  «|y  rare  in  American  English- -and  to  determine  the  text 
frequency  of  those  words.  Twenty  high  information  nouns  and  adjectives-- 10  rare 
singletons  and  10  keywords-  -  ./ere  selected  tor  use  as  question  words.  Singletons  are  high 
information  words  that  occur  only  once  in  a  passage;  and  keywords,  words  that  occur  more 
than  once. 

2.  Twenty  sentences  were  then  selected  for  transformation  into  multiple-choice 
items  by  four  item  writers.  Five  of  those  sentences  included  rare  singleton  nouns;  five, 
rare  singleton  adjectives;  five,  keyword  nouns;  and  five,  keyword  adjectives. 

The  stems  for  these  multiple-choice  items  were  produced  bv  substituting  the 
question  words  with  wh-words  (who,  what,  etc.).  For  example,  the  rare  singleton 
"silverfish"  appeared  in  the  following  sentence:  "The  most  primitive  insects,  such  as  the 
silverfish,  do  not  go  through  metamorphosis."  For  this  sentence,  one  wntei  produced  the 
following  item  stem:  "The  most  primitive  insects,  such  as  what,  do  ixit  go  through 
metamorphosis?"  Next,  for  each  of  the  20  stem  items  produced,  each  writer  produced 
two  sets  of  foils  or  alternatives.  One  set  was  produced  informally  by  the  writer;  and  the 
other,  bv  an  algorithmic  method.  For  example,  for  the  above  item  stem,  the  writer/au¬ 
thor  produced  the  following  foils: 

a.  Informally  - -Butterflies,  Silverfish,  Canine,  and  Cicadas. 

b.  AlgorithmicalIy--Silverfish,  Females,  Individuals,  and  Wasps. 

This  process  resulted  in  16.0  multiple-choice  items:  20  selected  sentences  trans¬ 
formed  by  four  item  writers  using  two  foil  methods.  For  a  given  instance,  the  stems,  as 
well  as  the  foils  produced  informally  bv  the  writers,  were  comparable  but  not  identical. 
The  foils  produced  algorithmically,  however,  were  the  same  across  items/writers. 
Examples  are  provided  in  the  appendix. 


To  generate  foils  for  tin'  rare  singleton  ami  key  word  nouns,  those  selected  as  question 
words  were  classified  semantically  using  the  method  developed  In  Frederu  ksen  (l*s2ST, 
which  is  shown  in  Figuie  I,  To  iMustiate,  using  Hus  method,  the  singleton  noun 
"silver tish"  would  he  classified  ns  a  i  om  note,  processivc,  .miniate  noun  (•'*  I V  Othei  rate 
singleton  and  keyword  nouns  in  ttie  passage  that  also  met  this  olassit  icat  uni  weie  then 
seU\  toil  at  random  to  create  toils.  Those  selected  as  toils  foi  "silverfish"  using  this 
method  were  "females,"  "individuals,"  and  "wasps,"  as  indicated  above. 
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Figure  1.  Fredericksen's  semant u  classification  of  nouns. 


To  generate  foils  for  tlie  adjective  quest  uni  words,  all  rare  singleton  and  keyword 
adpvtives  in  the  prose  passage  (not  hist  those  selected  as  question  words)  were  classified 
using  semantic  differential  techniques  (Nunnallv,  !%\  pp.  |n  research  using 

these  teihmques,  adjectives  are  tvpicalU  classified  based  on  their  U)  evaluation  (e.g., 
good  or  bad),  (2)  potenc\  (e.g,.  strong  or  weak),  (f)  activity  (e.g.,  fast  v'r  slow),  and  (•*♦) 
familiarity  (e.g.,  simple  or  complex).  In  addition  to  these  four  categories,  rare  singleton 
and  keyword  adjectives  in  tin'  prose  passage  were  classified  according  to  whether  01  not 
they  could  ho  considered  as  "technical"  words.  This  latter  category  is  part  ulurlv  useful 
m  technically  oriented  material,  particularly  tor  grouping  adjectives  that  relate  to  a 
certain  noun. 

■\fter  tliese  adjectives  were  classified  according  to  these  five  categories,  they  were 
analyzed  as  to  their  familiarity,  using  the  Hale-Cliatl  (l'*4S)  list  of  )000  familial  words.  If 
tliev  were  included  in  tiiat  list,  the\  were  not  used  as  foils  because  they  were  too  familial 
a  .1,  thus,  too  easy.  Approximately  ^0  adiectives  passed  this  screen  and  qualified  for  use 
os  foils.  From  this  group,  foils  were  developed  hv  randomly  selecting  those  having  tin' 


same  classification  as  tlx*  adjective  question  *ords  (i.e.,  as  to  elevation,  potency,  etc.). 
For  example,  thou*  selected  for  the  rare  singleton  "pupal"  were  "nymphal,"  "parasitic," 
and  "insect"  (see  appendix). 

From  the  160  items,  eight  20-itern  test  forms  were  developed.  Each  test  included 
five  items  generated  from  rare  singleton  nouns;  five,  from  keyword  nouns;  five,  from  rare 
singleton  adjectives;  and  five,,  from  keyword  adjectives.  In  addition,  test  forms  were 
organized  so  that  each  included  five  items  from  each  of  the  four  item  writers,  ten  .terns 
with  foils  generated  informally  by  the  item  writers,  and  ten  items  with  foils  generated 
algorithmically.  The  internal  consistency  reliability  estimates  (Kuder-Richardson  Reli¬ 
ability  Formula  Number  20)  averaged  .63  for  these  test  forms. 

The  eight  forms  were  administered  to  24  students  from  the  Oregon  College  of 
Education  before  (pretest)  and  after  (posttest)  they  had  studied  the  prose  passage  on 
insect  development.  For  both  pretest  and  posttest,  three  subjects  were  randomly  assigned 
to  each  of  the  eight  test  forms;  care  was  taken,  however,  to  ensure  that  the  pretest  and 
posttest  forms  administered  to  each  student  were  different. 

Average  pretest  and  posttest  item  difficulties,  as  determined  by  the  percentages  of 
students  who  answered  the  item  correctly,  were  computed  for  items  (1)  produced  by  each 
of  the  four  writers,  (2)  derived  from  each  of  the  four  types  of  question  words,  and  (3)  with 
foils  either  generated  informally  by  the  writers  or  algorithmically.  Also,  a  nonparametric 
analysis  of  variance  (ANOVA)  (Wilson,  1956)  was  used  to  examine  differences  in  item 
difficulties  between  (1)  the  four  item  writers,  (2)  the  four  question  word  types,  (3)  the  two 
foil  types,  and  (4)  the  two  test  occasions. 

Results  showed  that  items  based  on  rare  singleton  nouns  and  adjectives  and  keyword 
adjectives  showed  a  significant  change  in  item  difficulty  from  pretest  to  posttest, 
indicating  that  such  items  are  useful  in  learning  from  the  type  of  prose  used  in  the  study. 
Items  derived  from  keyword  nouns,  however,  produced  low  quality  items,  primarily 
because  the  sentences  they  occurred  in  were  usually  introductory  sentences  of  a  general 
nature. 

The  two  types  of  foils  proved  to  be  almost  equally  effective  for  learning,  as 
evidenced  by  the  similarity  in  posttest  item  difficulty.  Thus,  Roid  and  Finn  concluded 
that  the  methods  they  used  for  generating  foils  were  feasible.  Although  foils  produced  by 
these  methods  were  somewhat  easier  than  those  generated  by  item  writers,  they  still 
appeared  to  produce  a  significant  shift  in  difficulty  from  pretest  to  posttest  when 
instruction  was  provided  between  testing  sessions. 

Finally,  the  results  of  the  ANOVA  showed  a  strong  mean  effect  for  test  occasions, 
which  indicates  that  all  types  of  items  were  effective  for  learning.  There  was  also  a  main 
effect  for  word  type,  which  was  caused  by  the  easier  items  derived  from  keyword  nouns, 
as  noted  above.  Finally,  there  were  two  significant  three-way  interactions:  (1)  writers  bv 
word  type  by  pretest-posttest  and  (2)  writer:-  by  foil  types  by  pretest-posttest.  The  firsi: 
was  caused  by  variations  in  item  difficulties  in  items  produced  by  the  different  writers; 
and  the  second,  by  the  fact  that  one  writer  generated  better  foils  than  the  others. 

Purpose 

The  purpose  of  this  study  was  to  extend  or  replicate  the  Roid  and  Finn  study.  It  is 
expected  that  the  results  will  form  the  basis  for  additional  development  of  algorithmic 
procedures  for  generating  test  questions  from  prose  materials. 
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APPROACH 


Subjects 

The  eight  forms  developed  in  the  Roid  and  Finn  study  were  administered  to  249  high 
school  students  before  (pretest)  and  after  (posttest)  they  had  studied  the  passage  on  insect 
development.  For  bath  pretest  and  posttest,  approximately  30  subjects  were  randomly 
assigned  to  each  of  the  eight  test  forms.  Care  was  taken,  however,  to  ensure  that  the 
pretest  and  posttos;  forms  administered  to  each  subject  were  different. 

Analysis 


For  purposes  .it  analysis,  test  'vsults  from  the  earlier  study  were  combined  with  those 
obtained  in  this  study.  Thus,  the  totcl  number  of  subjects  was  273  (24  college  students 
and  249  high  school  students).  Since  the  number  of  subiects  reponding  to  each  test  form 
varied  from  27  to  38  on  the  pretest  and  from  23  to  33  on  the  posttest,  it  was  possible  to 
obtain  quite  stable  estimates  of  item  difficulties.  A  repeated-measures  analysis  of 
variance  (ANOVA)  (4x2x2x2x2  factorial  design)  was  used  to  examine  differences  in 
item  difficulties  between  (1)  the  four  item  writers,  (2)  the  two  parts  of  speech  (adjectives 
and  nouns)  of  question  words,  (3)  the  two  types  of  text  frequencies  (keyword  and  rare 
singletons),  (4)  the  two  foil  types  (writer's  choice  and  algorithmic),  and  (5)  the  two  test 
occasions  (pretest  and  posttest). 

With  160  items  given  on  two  occasions,  the  analysis  had  320  data  points,  and  five 
replications  per  cell.  The  ANOVA,  which  was  conducted  on  the  item  difficulties  for  items 
in  each  cell  of  the  design,  is  useful  for  determining  the  "instructional  sensitivity"  of 
item;-  A  significant  main  effect  for  the  pretest-pcsttest  factor  would  indicate  that 
pretest  difficulties  were  significantly  different  from  posttest  difficulties  for  all  items.  A 
significant  interac-'ion  effect  involving  the  pretest-posttest  factor  would  indicate  tnat 
certain  types  of  items  differed  in  the  pattern  of  their  pretest  and  posttest  difficulties. 


RESULTS  AND  DISCUSSION 


ANOVA  Results 


Table  1,  which  presents  the  results  of  the  analysis  of  variance  (ANOVA)  of  item 
difficulties,  shows  that  the  strongest  effect  was  the  main  effect  for  test  occasions  (R). 
This  finding  indicates  that,  across  all  types  of  items,  the  percentage  of  subjects  getting 
pretest  items  correct  was  lower  than  die  percentage  of  subjects  getting  posttest  items 
correct.  In  other  words,  most  items  showed  instructional  sens'tivity.  Table  2  shows  that 
pretest  item  difficulties  averaged  47.6  percent  across  all  items;  and  posttest  item 
difficulties,  74.4  percent.  This  indicates  that  the  subjects  did  lea_n  by  reading  from  the 
passage,  even  though  nearly  half  were  able  to  guess  the  correct  answer  to  most  questions 
on  the  pretest.  With  four-option  multiple-  choice  items  such  as  those  used  in  this  study, 
excellent  items  should  show  pretest  difficulties  nearer  to  the  level  of  random  guessing 
(25%). 

Two  important  findings  of  this  experiment  were  1  e  main  effect  of  part  of  speech  (P) 
and  ihe  interaction  of  P  and  the  repeated  measur  'RP),  as  shown  in  Table  l.  An 
inspection  of  Table  3--P  and  RP  interaction  effects--reveals  that  items  based  on  noun 
question  words  were  significantly  easier  overall  then  were  items  based  on  adjec- 
tives--65.6  vs.  56.3  percent.  Also,  the  difference  between  pretest  and  posttest 
difficulties  was  greater  for  nouns  than  for  adjectives  (29.5  vs.  24.1%)  (untabled),  which 
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Table  I 


Repented  Measures  Analysis  of  Variance  on  Item 
Difficulties  of  Items  of  llnch  Type 


Source 


dt 


I 


W  (Writers) 

I  (I  oil  Type) 

P  (Noun  vs.  Adjective) 

S  (Keyword  vs.  Rare  Singleton) 

WP 

WP 

PP 

WS 

rs 

PS 

WPP 

wrs 

WPS 

IPS 

WPPS 


' 

I 

I 

I 

I 

3 

I 

' 

I 

I 

' 

' 

\ 

I 

' 


Residual  128 

R  (Pretest  vs.  Post  test)  I 

RW  ' 

RP  I 

RP  I 

RS  l 

R  WP  ' 

R  WP  ' 

RPP  | 

RWS  ' 

RPS  I 

R  PS  I 

R  WPP  \ 

R  WPS  ) 

R  WPS  ' 

RPPS  I 

R  WPPS  1 

Residual  178 


17.4 

7.27 

. 

I .  (>t> 
.;7 
1.74 
I.  '3 
14.21* 
.  1 1 
.  34 
,2*> 

.  ! ' 

.  '4 


472.0'* 

1.40 

1.37 

4.70*** 
2. OS 
1  .  7  4 
7.  Vr 
'.04 
.4' 

4. 2S*  ' 
20.47  * 

I  .OS 
.el 
2.M 
.11 
.  V 


*p  <  .001. 

*p  <  .003. 
*  *  *p  <  .03. 


Table 


Moitns  and  Staiufatd  IVviattons  of  Item  Oitficulties 
on  Pietest  and  Post  tost 


Type  of  Itrm 

Pretest 

Mean 

s.n. 

Mean 

Post  test 

s.n 

Writer  (W): 

1 

4<>.S 

IS.  ' 

*1 .(. 

21.4 

2 

44.  S 

1 4 . 7 

77.7 

17.4 

1 

4(>.  1 

70 .  i 

*s,7 

70 . 2 

4 

4 ' . ; 

l*».r 

.  s 

1  7 . 7 

roll  (F). 

Writer's  Chou  e 

4<..  t 

70 . 1 

”4 . 

14.4 

Algorithmic 

48.8 

18.7 

*4.7 

18.4 

Part  of  Speech  (P): 

Noun 

M) .  8 

20 .  1 

80 . 4 

14.1 

Adjective 

44.  1 

18.7 

(.8 . 4 

21  .  S 

Stem  Type  (S): 

Keyword 

Ml.  4 

18.4 

* '  .4 

lc.4 

Rare  Singleton 

44.8 

70.0 

71.4 

71  .  ' 

Test  Forms: 

1 

47 .  ' 

17.  1 

*'>.  t 

70 . 4 

2 

48.  (' 

71.4 

SI.0 

1  7 .  (. 

1 

44.7 

717 

*7.0 

|4.4 

4 

M  .4 

18.4 

*4.0 

71.1 

% 

*>l .  7 

IS.  7 

*1.4 

1  (. .  7 

r> 

4t.  ,4 

I'.  1 

*7.4 

1  1.4 

; 

CM 

1 4 .  g 

(.7.4 

21.7 

s 

'1.1 

70 .  ' 

70 . 4 

21.2 

All  Items 

47.  (. 

14.4 

*4.4 

14.  | 

Table  1 


Means  and  Standard  IVviations  of  Item  Hif  f unities 
foi  Various  Interaction  llffcots 


Repeated  Measure  (R) 

Pretest  Post  test 

Average 

Variable 

Mean  S.IS.  Mean  S.H. 

Mean  S.O. 

P  and  RP  Interaction  Fffects 


Noun 

SO.  8 

20.1 

80. 1 

14.1 

6S .  6 

14.7 

Adjective 

44 .  1 

18.2 

68.4 

21 .  s 

S6.1 

18.1 

Average 

47.6 

19.4 

74.4 

I9.| 

61  .0 

17.  1 

PS  and  RPS  Interaction  Ilf  fee 

ts 

Noun-based  Item: 

Keyword 

61.1 

IS. 7 

81.  S 

12.8 

72.4 

12.1 

Rare  Singleton 

40.4 

18.6 

r.  i 

14.8 

SS .  8 

n.s 

Adjective-based  Item: 

K  evword 

19.4 

14.0 

07 . 4 

IS.  6 

SI.  4 

11.4 

Rare  Singleton 

49.1 

20.6 

69.4 

26.  1 

S9 .  1 

21.6 

Average 

47.6 

19.4 

74.4 

19.1 

<-!  .0 

17.1 

RFS  Interne 

tion  llffects 

Writer's  Choice  Foil: 

Keyword 

S2.S 

18.2 

7S.1 

17.  S 

68 . 9 

16.1 

Rare  Singleton 

40.2 

20.2 

74 . 0 

21.4 

S7.1 

17.9 

Algorithmic  Foil: 

Keyword 

48.2 

18.6 

7S.O 

IS. 4 

61." 

IS. 8 

Rare  Singleton 

49.1 

19.0 

72.7 

21.9 

61.0 

18.1 

Average 

47.6 

19.4 

74.4 

19.1 

61.0 

17.1 

Note.  See  Table  1  (or  definitions. 
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indicates  that  noun-based  items  had  greater  instructional  sensitivity  than  did  adjective- 
based  items. 

An  examination  of  the  PS  and  RPS  interaction  effects  in  Table  3  further  reveals  the 
source  of  the  difference  between  nouns  and  adjectives  in  this  study.  As  shown,  the 
average  difficulty  of  items  based  on  keyword  nouns  is  72.4  percent  compared  to  less  than 
60  percent  for  the  other  types  of  items.  This  is  because  keyword  nouns  typically  occur  in 
introductory  sentences  that  are  very  general  and  that  address  the  main  topics  o>  the 
entire  passage.  For  example,  in  the  passage  on  insect  development,  the  keyword  noun 
"insects"  appears  in  the  very  first  sentence,  which  happens  to  be  a  very  general 
statement--"The  life  of  most  insects  is  short  but  active."  Students  can  usually  answer 
questions  derived  from  this  type  of  sentence  without  having  to  read  the  prose  passage. 
Also,  keyword  noun  items  vote  relatively  easy  for  subjects  to  recall  on  the  posttest 
(average  item  difficulty  of  83.5%),  possibly  because  they  were  mentioned  several  times  in 
the  passage  (see  Table  4).  This  assumption  supports  Finn's  (1977)  hypothesis  that  the 
information  content  of  rar<  words  is  reduced  by  their  high  text  frequency.  Although  the 
fact  that  keyword  adjectives  produced  the  most  difficult  items  (53.4%)  appears  to  be 
inconsistent  with  that  hypothesis,  Table  4  shows  that  the  keyword  adjectives  occurred 
fewer  times  than  keyword  nouns.  Thus,  Finn's  hypothesis  does  apply,  in  that  higher  text 
frequency  was  related  to  the  easiness  of  items  constructed  from  keywords.  With  text 
frequencies  of  2  or  3,  the  keyword  adjectives  were  very  ciose  to  being  rare  singletons 


Table  4 

Question  Words  Selected  from  tin'  Passage 
and  Their  Text  Frequency 


Nouns 

Adjectives 

Rare  Singleton 

Keyword 

Rare  Singleton 

Keyword 

Instars 

Insect  (8) 

Plant-feeding 

Immature  (3) 

Cicadas 

Insects  ,20) 

Pupal 

Incomplete  (2) 

Silverfish 

Metamorphosis  (9) 

Spine-like 

Nymphal  (2) 

Wasps 

T:gg  (8) 

Self-made 

Aquatic  (2) 

Appetites 

Adult  (8) 

Worm- like 

Distinctive  (2) 

Note.  The  number  appearing  in  parentheses  behind  keywords  represents  text  frequency. 


The  rare  singleton  nouns  showed  a  good  pattern  of  pretest  and  posttest  difficulties. 
They  had  the  highest  average  instructional  sensitivity  -40.4  to  77.3  percent--a  difference 
of  36.9  percent.  The  rare  singleton  adjectives  were  somewhat  easier  on  the  pretest  and 
more  difficult  on  the  posttest  than  were  the  rare  singleton  nouns. 

As  shown  in  Table  1,  there  was  no  main  effect  for  writers  (W)  or  foil  type  (F),  nor  was 
there  a  significant  interaction  between  writers  and  foil  (WF).  This  result  is  somewhat 
surprising  in  that  different  writers  would  bo  expected  to  write  easier  or  harder  items 
when  they  were  allowed  to  choose  their  own  foils. 
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Table  1  does  show  one  interaction  (RFS)  involving  to"  pc.  The  means  and  standard 
deviations  of  item  difficulties  for  that  interaction  are  alsi  hided  in  Table  3.  As  shown, 
all  of  the  posttest  means  ate  very  similar.  A  Newman-Miels  a  posteriori  test  of  the 
differences  between  pietest  item  difficulties  in  this  interact  on,  Tiowever,  revealed  that, 
among  the  items  with  "writer's  choice"  foils,  the  rare-singletc  n  items  were  more  difficult 
on  the  pretest  than  were  the  keyword  items  (40.2  vs.  52.5%). 

Variance  Between  Writers 

The  variability  of  item  difficulties  across  item  writers  was  examined  to  determine 
whether  the  difficulties  of  items  constructed  with  "writer's  choice"  foils  varied  more 
across  writers  than  did  the  difficulties  of  items  constructed  with  algorithmic  foils.  It  was 
expected  that  some  writers  would  choose  very  difficult  foils  for  a  given  transformed 
sentence;  and  others,  easy  foils.  The  algorithmic  foils,  which  were  chosen  at  random  from 
matched  groups  of  similar  words  from  the  passage,  should  be  free  of  any  item-writer  bias, 
and,  hence,  less  variable  in  their  effects  on  item  difficulty. 

In  examining  the  variability  across  writers,  the  focus  was  on  each  sentence  that  was 
transformed  by  each  writer.  As  indicated  previously,  each  of  the  four  item-writers 
produced  multiple-choice  items  (stem  and  foil)  for  each  of  the  20  sentences  selected  for 
transformation.  It  was,  therefore,  possible  to  identify  four  item  difficulties  for  a  given 
combination  of  sentence  and  foil  technique.  For  example,  for  the  sentence  containing  the 
keyword  adjective  "immature,"  the  four  items  generated  using  the  "writer's  choice"  foil 
method  resulted  in  pretest  difficulties  of  38,  65,  52,  and  37  percent  respectively,  and 
posttest  difficulties  of  67,  63,  74,  and  52  percent.  The  pretest  and  posttest  variabilities 
were  then  calculated  across  these  item  difficulties,  as  shown  in  Table  5. 

After  all  variances  of  item  difficulties  across  writers  were  calculated,  they  were 
subjected  to  a  repeated  measures  ANOVA  in  which  the  dependent  variables  were  the 
natural  logarithms  of  the  variances  (Scheffd-,  1959,  p.  83).  The  design  for  this  analysis 
was  2  x  2  x  2  x  2  with  the  following  factors:  (1)  foil  type  (writer's  choice  vs.  algorithmic). 
(2)  part  of  speech  (noun  vs.  adjective),  (3)  stem  type  (keyword  vs.  rare  singleton  question 
word),  and  (4)  the  repeated  measure  (pretest  vs.  posttest).  Surprisingly,  results  showed 
that  there  were  no  significant  main  effects  or  interactions.  For  example,  even  though  the 
average  variability  of  the  writer's-choice  foil  method  was  115.31  percent  compared  to 
73.97  percent  for  the  algorithmic  foil  method,  the  differences  was  not  statistically 
significant. 

One  important  limitation  of  the  present  study  that  should  be  mentioned  is  that  only 
four  item  writers  were  employed.  Calculation  of  variabilities  across  only  four  writers  is 
clearly  susceptible  to  the  influence  of  any  one  of  the  four  item  difficulties.  With  a  larger 
sample  of  writers,  the  effects  may  have  been  more  clearly  detectable. 


CONCLUSIONS 

The  concept  of  using  a  computer-based  algorithm  to  analyze  prose  instructional 
materials  and  to  identify  high  information  words  (i.e.,  those  that  are  rare  in  American 
English)  appears  to  be  woikabie.  High  information  nouns  or  adjectives  identified  as  rare 
singletons  (those  occurring  only  once  in  a  passage)  arc  apparently  good  candidates  for 
question  words.  High  information  adjectives  identified  as  keywords  (those  occurring  more 
than  once  in  a  passage)  also  appear  to  be  good  candidates  for  question  words,  providing 
they  occur  only  two  or  three  times.  In  contrast,  keyword  nouns  apparently  are  not  good 
candidates,  particularly  when  they  occur  in  general  introductory  sentences. 
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Table  5 


Variabilities  and  Standard  Deviations 
of  item  Difficulties 


Item  Types 

Pretest 

Posttest 

Average 

Foil  Type: 

Writer's  Choice 

Var. 

HI.  V) 

101.21 

113.31 

s.n. 

11.46 

10.06 

10.74 

Algorithmic 

Var. 

69.94 

78.24 

73.97 

s.n. 

S.  36 

8.85 

8.60 

Part  of  Speech: 

Noun 

Var. 

94. 4  5 

S'). 63 

89.93 

s.n. 

9.72 

9.25 

9.48 

Adjective 

Var. 

97.  30 

92.47 

94.85 

S.D. 

9.86 

9.62 

9.74 

Item  Type: 

Keyword 

Var. 

102.19 

87.06 

94.32 

s.n. 

10.11 

9.33 

9.71 

Rare  Singleton 

Var. 

89 . 9  3 

90. 9  S 

90.44 

s.n. 

9.48 

9.  54 

9.  S| 

RECOMMENDATIONS 

1.  Rare  singleton  nouns  and  adjectives  and  keyword  adjectives  that  occui  infre¬ 
quently  in  instructional  material  should  he  used  to  select  sentences  from  prose  passages 
for  transformation  into  questions  that  measure  reading  comprehension.  Keyword  nouns 
should  not  be  used,  particularly  when  they  occur  in  general  introductory  sentences. 

2.  Methods  of  algorithmically  generating  foils  for  multiple-choice  versions  of 
sentence-derived  questions  should  be  further  refined  and  applied  in  a  variety  of  subject 
matter  areas. 
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APPENDIX 


THE  PROSE  PASSAGE  USED  IN  THE  EXPERIMENT 
AND  EXAMPLES  OF  ITEMS  PRODUCED  FROM  TEXT 


riiosi:  rASSAur  »>sn»  in  rur  i-xituimi'.n v 


INSF.CT 


Hu*  lit*'  i«f  most  niMVt\  is  »hoit  Imt  ,uti\«*  \  n\ 
trxx  lusts  ts  Ium*  a  Iih  span  ol  moic  than  .1  \  c‘»n 
>1  hit  span  NN  O  IIH'.III  tin*  llllli'  !•  m  n  Ill'll  tilt 
t-ee  »s  l.n*l  to  nn  hrn  tlu*  t tills  »lrNrlnp»*ti  *tlnlf  thrs 
I  rt  >  h*ok  ,it  ss  ll.it  h.ipprux  tliniue  1 1  ox  ptiitsl 
Ml  him  t  t>  tlr' flop  limit  reo  In  most  t.iM*s 
tlii  sr  rj»es  li.it*  li  * *1  itxitl**  tlif  Unix  ot  tin*  Inn, ill 
In  tlu  It  w  t,i\rx  111  wlmh  tin  ree**  hatrh  msulr 
tin*  Inn. ilr  tin*  xoune  ,itr  Knii  ultxr  Pii'm*  in 
ms  ts.  x i n  li  ,i>  tin'  iplmlx  ,nt'  sunl  tt'  l*r  \  in  ifMiN'iis 
\  \  '  ip  ,ili  i\in^ 

Insists  tll.lt  halt'll  1 1 1 till  t*|*C\  .lUt'l  tlli’N  It. in  r 
Is  1 11 1, ml  .nr  x.inl  t"  's  on  ip.nrux  vn|i  \  tp  ,ili  nix' 

N  |  oxt  insists  .lit*  t*\  tp.H * *1 1 x  hi  most  r.ist's  rarh 
ree  pio.hn  » x  .»  miii'Ii*  1m111.it  111  «*  nisri  t  l  Ionnon  n 
m  nitum  Npt  t  u  s  ot  p.u.iMtn  Nx.isps  v«*n  NttnhV 
tin*  t'\; %*  m.iN  pi mlt,*  t  Inno  01  nioir  n  onne 

Most  insist  t*v,o  lit*  Nt  iN  slist in*  tiN r  riu*  si:i*. 
sll.ipt  01  t*olm  lit  tlu*  t'CC  is  illth'll'llt  m  lin'st 
t  ,|st«s  h‘l  t'.irli  s|HSirs  ot  insist  rills  i’ll.lMi'x  ,1 
prison  nn  I  it  1  lus  m.nlr  .1  stmh  nl  »Im*m'  ree**  to 
nlontilN  tlir  uixts  t  tli.it  I, ml  tlu*in  almost  ,is  ranh 
.IS  ll  I.,  lu.l  stS'll  thr  .ul lilt 

Most  insist  t*cC'  nr  l.ml  m  ,1  pl.nv  tint  nniII 
pioN  nlr  »  tthri  piotivtion  ot  too*l  lot  tin*  Nonne 
I'i ott'i  tii'n  is  1  spts  i.ilK  ini|v«it,int  to  thoxr  insis  ts 
tli.it  on  t'l  nn  inti'i  in  tin*  ree  \taer  On  n  nx  min  me 
uitMiis  th.it  tin*  .nlnlt  insist  Kins  its  ree**  in  tin* 
l.»h  snnmu  1  01  imiK  t.ill  I  In  ree**  tln*n  .no  tloi 
main  until  tin*  m*\t  spurn*,  nn  hrn  1 1  it's  hat*  h  Most 
ot  tin  ulnlts  *'1  t lit*' *■  spts  u’s  ait*  killol  I'n  tin* 
ti  .*  host  lh‘NNt‘Ni  i  tin*  halt  lime  ot  tht'st*  '*n* i^s  m 
tin'  spurn:  pimlnrfx  iu'nn  iii«|in  nln.ils  tis  ram  on 
tin*  xprnrx 

Most  plant  Irtslmv;  lusts  t\ instim  tiM'h  laN  thi'ii 
oi;i*s  t'n  plants  that  tin*  n  outu;  tnsl  mi  Hus  m 
»  it  asi's  tin*  nnmatnit  insn  ts  ,  hains's  ot  shin  in  al 
It  this  tit  l«|  ot  no i*stn',atn*n  mtnrsts  \ on  tin*  stmh 
ami  pliotov;i.iplo  ot  insist  rees  might  tn.ikr  a 
mss  I  pinurt 

Vt  t  t'i  1  r.n  hum  tin1  pi  opr  1  st  ago  ol  tin  rlopim’iit. 
tin'  rev*  ss ill  hattli  Hu*  vomit;  insist  nn  usr  a 
iuuiiIh  i  ot  nnjn  x  to  e*'t  out  ot  tlu*  r£g  Somr  insis  ts 

If 


DF.YF.I.OPMF.NT 


*  Iu*nn  thru  nn  ,in  out  Othoi  s  I1.0  r  sjus  ial  vpmohkr 
stuirtinrs,  ralhsl  ree  hinxtns,  w  hu  h  v  lit  tlnoueh 
till*  *  Ill'll  Vhnr  .1(1'  souit'  ree**  nnIjuIi  li.or  sjusial 
NNrak  sjs'ts  in  thorn  l  In’  usnn;  insist  rsraprs 
liom  thrsi*  t  ithri  h\  n\ nv.yjme  m  h\  taking  in  an 
ano  luiivtmg  tlu*  slit'll  n\i?Ii  intotnal  pirssmr 


After  thr  fgg 


Mtri  hatt  lime  all  niMS’ts.  rwvpt  thr  most 
pinnitor  i»o  tluou\*li  a  srnus  (it  'tt’ps  in  th*N«*h*p 
limit  l  host*  sti't's  air  \-.illr*l  ftirMHinrp/nvn*  I  In* 
\Ni'i*|  nit‘tani''i  plmsis  tt'in.'s  tonn  tN\ . vius'k 
nn  t'l  ils  nu'ta.  uii'anim*,  t»»  thane'  ami  imupho 
im*ainne  toi in  lln  it  ton*  inrtamm pliosis  m.*aiis 
a  rhaner  in  loini  I'tns  rhai.e«*  in  toun  tssiiis  in 
t nn  *'  'hth  irut  nn  a n  s  l  lu'M  t nn  1  *  nn  a n  s  .in*  t  alhst 
ooniplrtr  an*l  nnoniplrti'  mrt.nnoi pluvMs  I  hr 
most  pinnitiNi*  insists  stuh  as  thr  sihmtish.  il»i 
not  1*0  tlnonvji  uu  t aim'i phosis  \\  hrn  tlu-N  liatrh 
thi'N  look  likt'  tln  ii  p. Hi  nts  m  torn  nn  .in  r\»s*pt 
that  tlu'N  an*  sinaMt'i  I'liru  ilt'N rh'pnit'nt  *s*nsists 
t't  eioN\niN|  laieoi  ami  Ussunme  ahlr  tt*  irpio 
llius* 


lmompl«*tf  Mft.imoi  ohosis 

Insis  ts  nn  In.  h  sIionn  ;tns  tNjs*  tit  mrtainoiphosu 
luNONonne  'N  Inrh  Kmk  '«*in  nun  h  hkt*  tin*  .ulnlts 
t'l  thr  spts  it's  llnsr  niunatiiir  insists  air  »  alksl 
mniphs  M  ith  tin*  t'Nix'ptimi  ot  mmiu*  a.piatn  sjs- 
no,  thr  pi  nn  1  pal  'litlt'irm  \  s  ht  lN\ mi  thr  to  mphs 
ami  a.hilts  ait'  in  si/i'  ami  f lit'  pii'st'in**  cl  Nviue> 
\srr  tllustiatnui  at  tin*  linin' 

\t>NN  think  h.n  k  to  thr  ih*\i  option  ot  thr  pin 
linn  to  nn  l">  h  nixivtx  hrlt'iie  V»fK».'po*Aj  Hrmrtn 
ht'i .  mu*  t*t  tin'  1  It.ii  at  tomtit  s  t't  thrsr  iminals  is 
a  liaol  t'iitt*i  tONrnne  .  alhsl  an  t  imM  h  ,'.**»  thr 
r  x  i"  k  r  1 1  ton  is  m.itlr  t'l  .1  nonlome  suhstams' 
ralusl  vhiltii  ki  tin'  v'hitiu  is  haol  ami  sthl  ami 
lias  n  n  n  httli'  stit  tth  lionlt*  tin*  t'xi'skrlrtou 
thro*  is  \riN  httlr  ns'iu  tt'i  eiosvtli 

In  onloi  t*»  eit'NN,  t In*  nx  niph  must  os,  apt'  this 
St' It  111.  ">>•  1  'iiscn  It  tK *t*s  this  In  stso'tiiu*  a  iu'nn 
t'Xtxskt'lt'lon  nmh'i  thr  t'l. I  tint*  \\  ht'u  tins  iu'nn 
skm  is  tsunph  tr  tin*  oM  skrlrtou  splits  iIonxii  tin* 
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Spoclal  I't'fmission  ;mt  .'.1  bv  b’lust  Ij'Svvt  Is  jliu  puMlsh.vl  bv 
\orox  K.liuat  ion  I’tibl  lo.tt  ions,  (o>  i')(*N  Xerox  tVrj'. 
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Kick  ami  the  insect  \vnlk\  away  ami  loaves  it  K* 
hind  Von  have  prokibls  myh  some  of  thc»c  dis 
carded  skins,  ciiMed  casts,  on  tree  trunks 

For  a  time  after  tin*  insis  t  discards  its  «>l« I  skin, 
the  new  ovuskelcton  is  soft  I'lns  Allow  %  the  c\u 
skeleton  to  expand  ami  nuke  tiom  lor  fmth<*i 
grow  th 

K.u  li  of  the  |tcnc«ls  between  molts  is  calh*d  an 
lrvrfiir  Some  nvinplts  go  tlricuglt  as  mam  as  eight 
01  moio  nwfais  U'foie  enraging  as  adults 
Aipiatie  s|hvu*s  that  undergo  mconr  di  ♦,  meta 
imirphswis  nn.st  go  through  one  mote  step  m  do 
\elopment  A*.  irsiiiphs  they  bieatlu  hr  means  ol 
gills  I  hese  gills  must  |*e  lephiccd  l»\  an  breath 
mg  oi  gaits  m  the  adult  stage  Tins  is  dime  in  the 
last  nvniph.il  uist.it  When  it  is  time  foi  the  adult 
to  etnetge,  the  nxiiiph  uses  to  the  sitrhur  and 
molts  The  fulls  dexelopal  adult  steps  out  of  the 
final  nvmpli.d  sktn  svitli  fulls  de\  eloped  oigans 
lor  breathing  air 

Complete  Melamotphosis 

Urn  is  the  tvpe  of  metamorphosis  that  most 
people  ate  familial  svitli  Butterflies  and  moths 
hast  complete  metamorphosis  Thc«e  ate  fon» 
distinct  stages  egg.  larva,  pupa,  and  adult  Since 
the  adult  s  mam  aotivits  is  pmdiuuig  eggs,  and 
I’m  sure  son  know  what  these  aie.  wc  "ill  spend 
our  time  stmlnng  the  larsa  and  pupa 

The  larvae’s  mam  |ob  m  hie  is  to  <  at  and  gnus 
Tlios  lias e  huge  appetites  l.anaeaie  w-n  dillei 
ent  limn  the  adults  Tlie\  sin  not  has  s’  compound 
eves  *% mgs,  ami  nsnalh  have  shewing  month 
pan  even  in  those  oialeis  svlieie  tlie  adults  have 
sucking  nis’iitli  parts 

\  Ian,’,  mas  continue  to  eat  ansi  gtnss  all  sum 
mer  \s  cold  ss  e  ither  approaches.  it  max  build  a 
swoon  ansi  pass  into  the  pupal  stage 

Most  ol  these  insects  pass  the  winter  inside  the 
cocoon  Because  no  as  tis  its  is  \  tsihle  at  tins  finis', 
the  pupa  has  Uen  fakels  sailed  a  Testing  stage” 
As  tnalls  a  great  deal  ol  astisits  is  going  on  Hie 
vounlike  Ian  a  is  changing  into  a  fulls  developed 
asliilt  \\  hen  tin1  weather  is  xsaim  again,  this  adult 
emerges  fioin  the  cocoon,  mates,  lavs  eggs,  ami 
starts  the  whole  proccx*  1  w ex  again, 

14 


Irl's  Gfl  Together 

\h>  t  nise*  is  lepiodnee  sevnalls  I  Ins  means 
that  to  liase  ergs  that  will  hatch,  a  male  uul  a 
lem.ih  'I  the  spe»  n  s  must  mate  I  he  <piestien  is 
I  low  do  the\  Inn  I  i  ,it  h  othei 

It  has  bt’t’n  known  hn  seals  that  senn  .  *t  the 
sounds  m.tdi  h\  t  in  kets  and  t  u  ad. is  ss  tie  a  t\  pe 
ol  nialiuv  *  ill  It  is  t  is\  to  see  how  (In  si  nise«  ts 
get  toe.il  ha  But  sshat  ahont  the  insists  tint  do 
1 1« *t  ni.ikr  noise  butt' lilies  loi  instant* 

It  h.  s  l»et  n  ilis»  os t  »t  d  that  tin  females  ol  these 
spmo  gist’  oil  a  distontise  odol  I  Ills  odoi  is 
ih  let  t.ihl*  bs  male  inset  ts  exit  git  *  dot. unis 
I  In  male  loiiox.%  Bn  .  st »  nt  1 1  .tit  kit  k  to  the  le 
iii.il* 

I  Ins  hum’s  to  iniiiil  an  mteiestmg  t  vpt  mm  nt 
son  1 1  m*.  ht  C  l  \  \  1 1  lend  ol  i  nine  ou«  e  »  aught  a  ie 

tenth  cmciged  lein.ih  I’lomethea  niotli  lie  put 
the  lem.ih’  in  .1  stieeii  t  age  and  set  it  outside  Ills 
window  in  less  than  two  honis  theie  wen*  tuou 
than  Jssents  males  lunging  on  tlie  ontsidi  ol  the 
tagi  W  I  is  tlon  t  sou  tis  tins  ss  ith  of  In  i  kinds  ol 
nise.  fs  It  xsoiild  makt  a  go  it  st  tent  e  pio|et  t 
Si  n  in  i’  has  uses  I  the  *  1  is.  ox  ei  \  ol  these  otloi  s  tt’ 
help  t  Imim.itt  limit  sn  ahlt  nisei  ts  It  was  t « u  i  nt  I 
l  hat  lem.ih  t  ot  k i  oat  lies  gas  t  t  >l|  ,io  atti  at  t  is  e  t*' 
malt  t  ot  kio.it  lies  odoi  V  tentists  luxe  been  able 
tt«  lepnxhit  i  tlussti’iit  uni  li.o  e  nsi  d  it  to  atti  at  t 
males  to  flaps 

l  rrmscs 

Hew  Dut  Von  Rp«\1* 

J  Name  anit  Urst  one  the  Cnee  Upes  at  tleveiopmrnt 
i use-  Is  ,  an  n'  fhtotiph 

What  ativ  i '  it  age  is  tlieie  m  nists  t  i’KKv  toeing  taut  on 
i  pit, tm  giants* 

5  V\h.»t  »s  mi'i.inun phiois ’  What  .ue  t’  tUffei I'liros 
between  t  cmp't’tc’  amf  mtoinpielt*  nit  .i.imi  I'ltt'sis* 

4  What  pus  esses  take  p»ate  tfwnnK  t Hi’  Knwsth  ol  m 

set  is’ 

V  Can  >on  fltnik  of  am  .nlvant.iKes  to  same  nise.  ts  in 
being  horn  alive  ' 

Read  A  tittle  More 

l  t  emmeo  S’  S  4a  ANu.t  Moths  an,f  Rt/flr  fin  s 
New  Xoik  Random  House  t9M» 
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Not e.  Special  permission  granted  by  What  Insect  1 s 
hv  Xerox  Education  1’tib  1  lc.it  Ions , 


_ _  That  ?  pu  Mi  shed 

(c)  19hS  Xerox  t'orp. 
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EXAMPLES  OF  ITEMS  PRODUCED  FROM  TEXT 


Keyword  Noun — Metamorphosis. 

a.  Text  Sentenee(s):  After  hatching,  all  insects,  except  the  most  primitive, 

go  through  a  series  of  steps  in  development.  These 
steps  are  called  metamorphosis. 

b.  Items  (Stem  and  Foils)  Produced  hv  Item  Writers: 

(1)  What  are  the  series  of  steps  in  insect  development  called? 

(a)  Maturation  (c)  Symbiosis 

(b)  Metamorphosis  (<))  Meitosis 

(2)  What  are  the  steps  insects  go  through  in  development  called? 

(a)  Metamorphosis  (c)  Larva 

(b)  Arthropoda  (d)  Puna 

(3)  What  are  a  series  of  steps  in  development  called? 

(a)  Reproduction  (c)  Metamorphosis 

(b)  Larvae  (d)  Changes 

(4)  What  are  the  series  of  steps  in  insect  development  called? 

(a)  Encrytid  (c)  Arthorpoda 

(b)  Instar  (d)  Metamorphosis 

c.  Foils  Produced  Algorithmically: 

Growths 

Metamorphosis 

Types 

Activities 

Rare  Singleton  Noun — Silverfish. 

a.  Text  Sentence:  The  most  primitive  insects,  such  as  the  silverfish,  do 

not  go  through  metamorphosis. 

b.  Items  (Stem  and  Foils)  Produced  by  Item  Writers: 

(1)  What  does  not  go  through  metamorphosis?  The 

(a)  Moth  (c)  Nymphs 

(b)  Silverfish  (d)  Butterfly 

(2)  What  do  not  go  through  metamorphosis?  The  most  primitive  insects, 
such  as 

(a)  Silverfish  (c)  Spiders 

(b)  Termites  (d)  Moths 

(3)  What  insects  do  not  go  through  metamorphosis?  The  primitive,  such  as 

(a)  Eggs  (c)  Chitin 

(b)  Silverfish  (d)  Butterflies 


(4)  The  most  primitive  insects,  snob  .is  what,  do  not  go  through  metamorphosis 

(a)  Butterflies  (c)  Canines 

(b)  Silvoifisli  (it)  Cicadas 

i- .  toils  Produced  Algori  ..nicallv: 

S  i  1  vor  f  tsli 
Fenw  l  os 
Liul  iv  i  itua  l  s 
Wasps 


1.  Keyword  Adjective — Immature. 

a.  Text  Sentence:  In  most  oases,  each  egg  produoes  a  single  iroront ure  insoot  . 

b.  Items  (Stem  and  Foils)  Prodnood  by  Item  Writers: 

(l)  What  does  each  egg  produce  in  most  cases?  A  single 

(a)  Immature  insect  (c)  Adolescent  insect 

(b)  Adult  Insect  (d)  Mature  insect 

(?)  What  does  each  egg  produce  in  most  cases?  A  single 

(a)  Oviparous  insect  (c)  Mature  insect 

(b)  Nvmphal  Insect  (d)  Immature  insect 

(!)  lu  most  cases,  what  does  each  egg  produce?  A  single 

(a)  Dormant  Insect  (c)  Adult  insect 

(b)  Adult  insect  (d)  Immatuie  insect 

(4)  What  does  each  egg  produce?  A  single 

(a)  Immature  insect  (c >  Round  insect 

(b)  Mature  uhsect  (d)  Adult  insect 

c.  Foils  Produced  Algor l thmlcal 1 v : 

Complete  insect 
Distinct  insect 
Immature  insect 
Incomplete  insooi 

>t .  Rare  Singleton  Ad  ject  ive— Pup_al  . 

a.  Text  Sentence (s) :  A  larva  may  continue  to  eat  and  grow  all  summer.  As 

cold  weather  approaches,  it  may  build  a  cocoon  and 
pass  into  the  jnijvtl  stage. 

b.  Items  (Stem  and  Foils)  Produced  by  Item  Writers: 

(l)  What  may  a  larva  do  as  the  cold  weather  approaches?  Build  a  cocoon 
and  pass  into  the 

(a)  Nvmphal  stage  (c)  Piipal  stage 

(b)  Parasitic  stage  (d)  Molt  stage 
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(2)  As  cold  weather  approaches ,  a 


into  what? 

(a)  Infant  stage  (e) 

(hi  Adult  stage  (d) 

(1)  Into  wliat  st ape  wav  the  larva 
it  builds  a  cocoon.  The 

(a)  larval  stage  (c) 

(c)  Pupal  stage  (d) 


larva  may  build  a  cocoon  and  pass 

Butterfly  stage 
l’u  pa  1  stage 

pass  as  cold  weather  ipproachcs  and 

Skeletal  stage 
Nvmphal  stage 


(4)  As  cold  weather  approaches,  what  niav  a  larva  do?  Build  a  cocoon 
and  pass  into  the 

(a)  1’npa l  stage  (e)  Dormant  stage 

(hi  Hibernation  stage  (d)  Resting  stage 


Foils  Produced  Algorithmically: 

Pupa  1  stage 
Nvmphal  stage 
Parasitic  stage 
Insect  stage 
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